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What  type  of  analysis  should  be  used  to  inform  design  when  both  the  future  organizational  structure  and  the  experimental 
simulation  are  tenuous?  Neither  the  application  of  a  qualitative  process  tracing  method  nor  low-level  quantitative 
organizational  designs  are  warranted.  We  hypothesized  the  analyses  of  high-level  communication  patterns  in  a  role-playing 
exercise  of  a  future  organization  structure  would  yield  results  that  could  both  inform  organizational  design  and  shape  iterative 
experimental  designs.  This  study  summarizes  the  comparison  between  communication  patterns  in  an  envisioned 
organizational  structure  and  the  actual  patterns  of  information  exchange  of  experienced  military  participants  role-playing  staff 
members  in  a  future  organizational  design.  The  comparison  between  the  hypothesized  and  actual  communication 
performance  indicated  a  different  distribution  of  communication  interaction  from  the  expected.  These  results  help  guide  both 
the  future  organizational  concept  as  well  as  next  iteration  experiments. 

over  three  days  (8+  hours  of  coding).  The  coding  and 
comparison  was  not  conducted  “in  the  blind”.  However,  as 
will  be  explained  in  this  paper,  that  level  of  experimental 
method  is  not  required  at  this  stage  of  the  experimental- 
organizational  design  iteration  process. 

The  purpose  of  this  comparison  is  to  examine  an 
alternative  initial  high-level  quantitative  approach  to 
qualitative  and  low-level  quantitative  approaches. 
Qualitative  approaches  such  as  future  incident  forecasting 
(Smith  et  al,  1998)  or  process  tracing  methods  (Woods, 
1993)  may  be  inappropriate  because  of  a  lack  of 
specification  of  the  organization  concept.  For  the  same 
reason,  low-level  quantitative  measures  such  as  those 
provided  by  organizational  design  systems  (Entin  &  Entin, 
2001)  are  also  unsuitable.  However,  a  gross  level 
comparison  of  high-level  quantitative  measures  such  as 
communication  allows  for  generation  of  specific  research 
questions,  measurement  methodologies,  and  metrics  to  be 
designed  into  the  next  iteration  of  the  design  process.  In 
addition  to  designing  future  iterations  of  the  experiment, 
this  high  level  quantitative  analysis  identifies  Objective 
Force  issues  via  an  envisioned  vs.  actual  performance  delta. 
This  delta  can  be  explained  through  cognitive  systems 
engineering  issues  (human,  technology,  training,  etc)  that 
can  be  addressed  in  the  next  iteration  of  organizational 
design.  We  propose  that  qualitative  and  a  more  specified 
quantitative  set  of  metrics  will  be  useful  once  future 
iterations  of  this  organizational  design  and  experimental 
design  have  reached  a  greater  level  of  specification. 


INTRODUCTION 

The  US  Army  is  in  the  opening  phase  of  a  ten-year 
organizational  design  process  for  a  knowledge-centric 
command  and  control  element.  In  support  of  this  initial 
effort,  the  Fort  Leavenworth  Battle  Command  Battle 
Laboratory  (BCBL)  has  conducted  the  first  high  fidelity 
experiment  to  determine  organizational  constructs  that  will 
support  command  and  control  in  the  Objective  Force.  The 
experiment  assumed  a  knowledge-centric  staff  cell  structure 
supported  by  a  higher  level  of  automation  to  increase 
shared  situation  awareness.  The  automation  specifically 
assists  commanders  and  staff  in  detecting  and  resolving 
short-term  conflicts  in  real-time.  The  advent  of  this 
capability  implies  that  the  force  will  not  be  as  dependent  on 
tactical  and  strategic  constraints  during  battle  planning  to 
ensure  the  smooth  execution  of  operations.  This  also  means 
that  the  Command  and  Control  relationships  and 
communication  requirements  will  be  different  from  those  in 
today’s  operational  command  and  control  structure. 

This  study  summarizes  findings  that  compared  the 
envisioned  communications  performance  against  the  actual 
communications  performance  of  experienced  military 
members  role-playing  staff  members  in  the  future  Objective 
Force.  The  envisioned  communications  performance  metric 
was  developed  via  a  complex  knowledge  object-grouping 
task  by  an  expert  panel  of  military  officers  and  Army 
scientists.  The  experimental  measures  of  communication 
performance  were  derived  from  the  data  log  of  three 
offensive  battle  simulations  conducted  by  the  role-players 
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METHOD 

Envisioned  Communication  Performance 

A  representative  set  of  Army  intellectuals  was  assembled 
at  Ft  Leavenworth  six  months  before  the  command  and 
control  experiment  Twenty-six  Army  officers  ranging  in 
rank  from  Brigac  General  to  Captain1  served  as 
participants  in  knc..,_dge  object  (KO)  development  and 
grouping.  All  participants  had  worked  with  the  military  for 
a  minimum  of  7  years  and  the  average  time  working  with 
the  Army  was  approximately  16  years. 

Participants  were  given  one  day  of  military  decision¬ 
making  training  (review)  to  provide  common  ground  for 
discussion  in  future  sessions.  On  day  two,  in  four  ninety 
minutes  sessions,  the  group  specified  a  separate  set  of  KOs 
for  each  of  the  typical  operational  missions  of  an  Army 
command  and  control  element:  Offense,  Support,  Defense, 
Stability.  They  then  individually  submitted  ratings  of  each 
of  the  knowledge  objects  to  specific  operational  missions  as 
in  Figure  1.  On  day  three  all  participants  were  assembled 
and  required  to  create  “Natural  Knowledge  Clusters”  based 
on  mission-type  for  each  of  the  knowledge  objects. 


Figure  1.  Knowledge  Object  development,  prioritization  and 
clustering  resulted  in  cell  structures  designed  for  Combat 
(offensive,  defensive),  Support,  and  Stability  operations. 


Knowledge  object  ratings  were  distributed  using  a 
majority  wins  process.  The  “Natural  Knowledge  Clusters” 
resulted  in  a  proposed  cell  structure  for  the  experimental 
staff  organization.  The  knowledge  objects  were  then 
clustered  a  second  time  to  maintain  a  consistent  command 
and  control  organization  across  operational  missions.  The 
resulting  experimental  staff  organization  and  the  knowledge 
object  distribution  is  represented  in  Figure  2. 


representing  13  military  posts  and  16  command 
organizations  and  nine  Army  and  civilian  scientists 
representing  an  equal  number  of  theoretical  approaches 


Figure  2.  Cell  structure  resulting  from  initial  Knowledge 
Object  distribution  workshop  conducted  at  the  Ft  Leavenworth 
Battle  Command  Battle  Laboratory.  1 1 1  of  the  knowledge  objects 
are  represented  in  this  depiction. 

Actual  Communication  Performance 

In  February  2003,  the  first  experiment  iteration  using  the 
knowledge  object  cell  structure  was  conducted  at  the  Fort 
Leavenworth  BCBL.  26  Army  officers  served  as  role- 
players  for  the  experimental  command  and  control  staff. 
They  spent  one  week  in  team  training.  During  this  time, 
they  were  required  to  learn  a)  the  concepts  behind  the 
experimental  organization,  b)  a  new  method  to  make 
decisions  in  the  experimental  organization,  c)  their  role  in 
the  structure  of  the  experimental  organization,  and  d)  how 
to  use  the  simulation  software  during  the  experiment. 

The  experiment  itself  was  conducted  as  a  command  post 
exercise  using  OneSAF  simulation  software.  The  role- 
players  gathered  information  and  input  actions  on  the 
battlefield  via  the  simulation.  Throughout  the  experiment, 
an  average  of  fourteen  data  collectors  entered  observations, 
real-time,  into  a  data  log  on  Group  Systems  (Nunamaker  et 
al,  1991).  As  a  result  every  observable  information 
transaction  was  captured.  Each  entered  transaction  was 
time-stamped  with  the  participants,  the  discussion,  and  the 
outcome. 

We  coded  the  data  log  for  speaker  /receiver,  prompted/ 
unprompted  push/pull  of  information,  purpose  of 
interaction  (military  context  relevant  categories).  Due  to  a 
limitation  of  audio,  we  were  only  able  to  check  the  data  log 
entries  against  the  commander’s  (Command  Integration 
Cell,  Figure  2)  position.  Therefore,  we  only  coded  entries 
in  which  the  commander  was  either  the  speaker  or  the 
receiver.  Further,  we  only  coded  the  first  three  days  of 
operations.  All  three  of  these  simulations  were 
progressively  more  complex  offensive  operations. 
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RESULTS 

The  distribution  of  communications  to  and  from  the 
commander  position  from  the  data  log  codings  yielded  an 
unexpected  result.  Specifically,  about  60%  of  the 
commander’s  interactions  were  outside  his  command  and 
control  organizatioi  almost  all  of  those  interactions  were 
to  his  subordinate  li  rs  in  the  operation  (See  Figure  3). 


Figure  3.  Envisioned  (Desired)  knowledge  object  distribution 
moderated  communications  compared  to  the  actual  leader 
(commander)  communications  during  the  first  experiment 


While  this  was  an  interesting  finding,  we  also  had  to  look 
into  the  percentages  of  knowledge  object  calculated  from 
the  distribution  previously  shown  in  Figure  2.  The 
distribution  of  communications  to  and  from  the  commander 
is  shown  in  Figure  4.  The  resulting  percentages  support  an 
envisioned  prioritization  of  leader  (commander)  interaction 
with  his  staff:  Command  Integration  Cell,  Information 
Superiority  Cell,  Maneuver  &  Support  Cell,  Fires  &  Effects 
Cell,  followed  by  the  Build  and  Sustain  Cell.  Even  after 
model  adjustments  to  account  for  external  leader 
communication,  every  actual  leader-cell  interaction  from 
the  data  log  codings  differed  from  the  envisioned  leader¬ 
cell  interaction  from  the  knowledge  object  distribution.  As 
opposed  to  the  envisioned  communications,  the  actual 
communications  priority  favored  the  Effects  Cell.  The 
comparisons  were  similar  in  their  low  valuation  of 
communication  with  the  Logistics  (Build  and  Sustain)  Cell. 

DISCUSSION 

There  are  two  major  quantitative  deltas  in  the  actual  vs. 
envisioned  interactions.  First,  the  amount  of  leader 
communication  external  to  the  staff  organization  was  about 
60%  of  his  total  communication.  Second  the  actual 
distribution  of  interaction  within  the  staff  did  not  match  the 
envisioned  distribution  of  interaction  based  on  the 
knowledge  object  assignments.  These  two  findings  have 
major  implications  for  both  next  iteration  experimental 
design  as  well  as  the  design  of  the  future  organization.  The 
remainder  of  this  section  will  propose  adjustments  that  will 
reduce  the  delta  to  insignificance  before  final  organizational 
design  implementation  and  fielding. 


“Desired"  Knowledge  Object  Distribution  (BCBL  Workshop) 
vs  FROM  Commander  Communication  (BCBL  Experiment) 


Staff  Cell 

Table  1.  Envisioned  prioritization  of  the  leader-cell 
communications  are  based  on  the  knowledge  object  distribution. 

The  next  iteration  of  experiments  must  establish  measures 
to  capture  process  data  between  the  leader  and  the  external 
communicators.  The  process  measures  will  better  explain 
potential  organizational  impacts  of  the  external 
communications.  In  the  case  of  the  organizational 
structure,  we  will  also  have  to  reconsider  the  role  of  the 
subordinate  commanders  and  their  knowledge  object 
responsibilities.  This  change  should  increase  the  validity  of 
next  iteration  experiments. 

The  knowledge  objects  distribution  contributes  to  a 
second  problem.  The  simulation  allows  members  full 
access  to  all  data  and  does  not  provide  a  prioritization 
system.  Shattuck  et  al  (2000)  hypothesized  that  access  to 
all  data  was  counter-productive  to  leader  decision-making. 
To  replicate  a  better  technological  support  to  decision¬ 
making  in  the  future  organization,  the  salience  of 
knowledge  objects  should  be  increased  based  on  context. 
The  next  knowledge  object  grouping  task  will  need  a 
greater  level  of  specification  and  a  “push”  system  is 
necessary  before  the  next  iteration  experiment. 

The  presence  of  a  large  shared  visual  display  presenting 
all  available  information  may  have  contributed  to  the  delta. 
In  their  study  of  shared  mental  models,  Bolstad  &  Endsley 
(1999,  2000)  found  that  providing  a  fully  shared  display 
was  actually  a  detriment  to  performance.  Teams  that 
started  with  a  fully  shared  visual  display  never  reached 
performance  levels  of  teams  with  a  partial  or  no  shared 
visual  display.  This  research  indicates  it  is  possible  that  the 
presence  of  a  large  screen  shared  visual  space  was  counter¬ 
productive  to  some  organizational  communication.  The 
next  iterations  of  the  future  organization  experiment  will 
examine  the  value  of  this  technology  to  group  processes. 

Bolstad  &  Endsley  (1999,  2000)  found  that  training  a 
shared  mental  model  would  improve  performance.  There  is 
some  evidence  that  the  role-players  often  developed  an 
inappropriate  shared  mental  model  based  the  decision¬ 
making  heuristics  from  their  years  of  legacy  organization 
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experience.  One  week  of  classroom  instruction  was  not 
sufficient  for  them  to  develop  future  organization  thinking 
and  heuristics.  One  heuristics  training  approach.  Instance 
Based  Learning,  is  under  development  by  Gonzalez  et  al  (in 
press).  The  approach  holds  promise  as  a  pre-simulation 
training  methodology  for  role-players  in  future  simulations 
and  could  provide  '  ippropriate  heuristic  base  and  shared 
mental  model  at  the  leriment  outset. 

CONCLUSION 

These  experiments  are  extremely  expensive.  Therefore 
sample  size  will  remain  small.  As  such,  it  is  difficult  to 
give  validity  observations  that  occur  in  a  small  number  of 
replications  in  the  early  stages  of  a  design  process. 
However,  at  this  first  iteration  of  a  decade  long  process,  we 
found  that  a  high  level  quantitative  approach  was 
appropriate.  Describing  the  delta  obtained  from  a  high 
level  quantitative  approach  using  informed  observation  and 
established  human  computer  interaction  theory  provided 
significant  findings  that  will  shape  both  the  organizational 
design  and  the  experiment  methodologies. 
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